Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[Flaky test] Skip test_operator_gpu.test_convolution_independent_gradients #15631

Merged
merged 6 commits into from
Jul 26, 2019
Merged

[Flaky test] Skip test_operator_gpu.test_convolution_independent_gradients #15631

merged 6 commits into from
Jul 26, 2019

Conversation

zixuanweeei
Copy link
Contributor

@zixuanweeei zixuanweeei commented Jul 23, 2019

Description

As it was descibed in #15603, test_convolution_independent_gradients has occasional assertion error on gpu device. This PR intends to skip the unit test when ctx.device_type="gpu". @pengzhao-intel @TaoLv @ciyongch

Checklist

Essentials

  • Changes are complete.
  • Tests on GPU device.

Changes

  • Skip the unit test on GPU device.
  • atol = 1e-3, rtol = 1e-3 by default.

Comments

  • An issue required to be filed to the community after collating some reproduced examples.

# set a low bar for autotuned cudnn conv
atol = 1.0e-1 if ctx.device_type == "gpu" else 1.0e-3
rtol = 1.0e-2 if ctx.device_type == "gpu" else 1.0e-3
if ctx.device_type == "gpu":
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can set ctx = mx.cpu() explicitly to avoid testing it on GPU. Please also add description or issue link so we can revisit it in the future.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. I will add some notes after filing an issue.

@abhinavs95
Copy link
Contributor

@mxnet-label-bot add [test, Flaky]

@zixuanweeei
Copy link
Contributor Author

@TaoLv Please take some reviews on this PR again. Thanks.

@TaoLv TaoLv merged commit 5e6ba7b into apache:master Jul 26, 2019
@zixuanweeei zixuanweeei deleted the gpu-ut-off branch July 26, 2019 08:36
juliusshufan pushed a commit to juliusshufan/incubator-mxnet that referenced this pull request Aug 12, 2019
…ients (apache#15631)

* Skip test_convolution_independent_gradirents

* Add an issue link

* Fix inconsistent context of input array and binding op

* Trigger CI

* Retrigger CI
juliusshufan pushed a commit to juliusshufan/incubator-mxnet that referenced this pull request Aug 12, 2019
…ients (apache#15631)

* Skip test_convolution_independent_gradirents

* Add an issue link

* Fix inconsistent context of input array and binding op

* Trigger CI

* Retrigger CI
TaoLv pushed a commit that referenced this pull request Aug 13, 2019
…o weights… (#15805)

* [MKLDNN] Independent gradients requests check with respect to weights and bias of convolution (#15497)

* Independent req[kBias] and req[kWeight] check

* Add UT for independent conv gradient requests

* Update conv independent grad UT with no_bias enabled

* Check req[kWeight] for avoiding unnecessary prim registration

* Check `OpReqTpye` in CommitOutput automatically

* Lock cudnn autotune for accurate conv output

* Ignore independent gradients test on GPU

* Trigger CI

* Sets a low bar for autotuned cudnn convolution

* [Flaky test] Skip test_operator_gpu.test_convolution_independent_gradients (#15631)

* Skip test_convolution_independent_gradirents

* Add an issue link

* Fix inconsistent context of input array and binding op

* Trigger CI

* Retrigger CI
anirudhacharya pushed a commit to anirudhacharya/mxnet that referenced this pull request Aug 20, 2019
…ients (apache#15631)

* Skip test_convolution_independent_gradirents

* Add an issue link

* Fix inconsistent context of input array and binding op

* Trigger CI

* Retrigger CI
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants